Eagle: A Better Hybrid Data Center Scheduler

Eagle is a new hybrid data center scheduler that considerably improves the job completion times for short jobs. Eagle builds on the Hawk hybrid scheduler, using a centralized scheduler for long jobs and distributed schedulers for short jobs. The main innovation in Eagle is that it provides an approximate and potentially slightly outof-date summary of the centralized scheduler state to the distributed schedulers to guide them in their scheduling choices. This approach leads to much better job completion times for short jobs, compared to the random probing and work stealing used in Sparrow and Hawk. The challenge is how to provide this information to the distributed schedulers without negatively impacting the performance of the centralized scheduler, and therefore the job completion times of the long jobs. The approximate nature of the information provided to the distributed schedulers is the key to an efficient implementation. The information is piggybacked on task assignment messages by the centralized scheduler, and eventually forwarded to the distributed schedulers. In particular, our implementation does not require the centralized scheduler to send any additional message, and only requires a very modest amount of computation and additional data to be sent. We evaluate Hawk by trace-driven simulation and a prototype implementation in Spark. Using the popular Google, Yahoo and Cloudera production traces, Eagle improves 50th, 90th and 99th percentiles of the short jobs completion times distribution by a factor of 1.72 on average, and up to 8.33, with a negligible impact on long jobs completion times (< 1% slowdown on average).